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SUMMARY 

Let  S  be  a  physical  system  whose  state  at  any  time  Is  des¬ 
cribed  by  an  n-dlmensional  vector  x(t),  where  x(t)  Is  determined 
by  a  linear  differential  equation'J.-  Az,  with  A  a  constant 
matrix.  Application  of  external  influences  will  yield  an  In¬ 
homogeneous  equatlon,"2£  *»  Az  +  f,  where  f,  the  "forcing  term", 
represents  the  control.  A  problem  of  some  Importance  In  the 
theory  of  control  circuits  Is  that  of  choosing  f  so  as  to  reduce 
z  to  0  In  minimum  time.  If  f  is  restricted  to  belong  to  the 

h ) 

class  of  vectors  whose  i~—  components  can  assume  only  the  values 
♦bjf,  the  control  is  said  to  be  of  the  'bang-bang"  type. 


Various  aspects  of  the  above  problem  have  been  treated  by 

M 

McDonald,  Bushaw,  LaSalle  and  Rose.  Ms  shall  consider  here  the 
case  where  all  the  solutions  of  =  Az  approach  zero  as 
In  this  case  we  prove  that  the  problem  of  determining  f  so  as  to 
minimize  the  time  required  to  transform  the  system  into  the  re3t  , 
position  subject  to  the  requirement  that  f^,  the  1— -  component, 
satisfies  the  constraint  |  f ^  |  b^  may  be  reduced  to  the  case 

where  f^  +b^ .  Furthermore,  we  show  that  if  all  the  charac¬ 
teristic  root3  of  A  are  real  and  negative,  f^  need  change  value 
only  a  finite  number  of  times  at  most,  dependent  upon  the  dimen¬ 
sion  of  the  system. 


Finally,  an  example  Is  given  for  n  r-  2,  illustrating  the 
procedure  that  can  be  followed  and  the  results  that  can  be  ob¬ 
tained. 

/ 
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ON  THE  "BANO-BANa"  CONTROL  PROBUSW 

by 

Richard  Bellman,  Irving  Olicksberg  and  Oliver  Oross 
§1 .  Introduction 

Let  z  be  an  n-dlmensional  vector  function  of  t  satisfying  the 
linear  differential  equation 

j}f  *  az  +  r.  z(o)  -  c  (1.1) 

where  we  assume  that: 

a.  A  is  a  real,  conatant  matrix  of  order  n,  whose  charac¬ 
teristic  roots  all  have  negative  real  parts; 

b.  f  is  reatrlcted  to  be  real,  measurable,  and  to  have  com¬ 
ponents  satisfying  the  constraints,  [ f ^  |  1. 

The  first  condition  is  the  necessary  and  sufficient  condition 
that  all  the  solutions  of  (l.l)  approach  zero  as  t — ^oo. 

The  problem  we  wlBh  to  consider  is  that  of  determining  the 
vectors  f  which,  subject  to  the  constraint  (b),  reduce  z  to  zero 
in  minimum  time.  This  is  a  problem  of  Bolza  of  rather  unconven¬ 
tional  type,  and  the  techniques  we  shall  employ  are  quite  different 
from  the  classical  ones. 


We  shall  establish  two  results: 
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Theorem  1.  Under  the  above  conditions,  an  f  which  reduces  z  to 
zero  In  minimum  time  exists,  and  has  components  f^  for  which  (f^l  - 

Theorem  2.  If  the  characteristic  roots  of  A  are  real,  diet  '  net, 
and  negative,  a  minimizing  f  exists  with  components  f^  for  which 
|fjJ  =  1,  and  each  f^  changes  sign  at  most  (n— 1)  times . 

The  statement  In  Theorem  1  has  been  assumed  in  the  paBt  on 

an  Intuitive  basis,  see  McDonald,  [3],  and  has  been  established  In 
various  cases  by  Bushaw,  [l] ,  LaSalle,  [2],  and  Rose,  [4].  The 

only  paper  we  have  had  access  to  is  that  by  Rose,  and  his  methods 

are  distinct  from  ours.  In  addition,  he  13  primarily  Interested 
In  the  case  where  the  condition  in  (a)  13  not  satisfied. 

Problems  of  this  type  arise  in  connection  with  many  differ¬ 
ent  types  of  control  processes.  A  discussion  of  the  connection 
with  servomechanisms  is  sketched  In  [2]  . 

§2.  Proof  of  Theorem  2. 

We  shall  consider  In  detail  only  the  case  of  Theorem  2, 
where  the  characteristic  roots  of  A  are  real  and  negative.  It 
will  be  clear  from  the  treatment  of  this  case  how  the  proof  of 
Theorem  1  goes. 

Let  X  be  a  square  matrix  whose  columns  are  the  n  linearly 
Independent  eigenvectors  x^  of  A,  and  let  (j  =  l,...,n)  be  the 
corresponding  n  distinct,  negative  eigenvalues  of  A;  clearly, 

X  is  non— singular  and  all  Its  elements  are  real.  Finally,  denote 
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by  A  the  diagonal  matrix  whose  J—  diagonal  element  i3  X^.  We 
have 


Ax 


J 


Vj 


(2.1) 


whence  we  see  that  AX  -  XA  ;  hence 


X_1AX  -  A 


(2.2) 


If  now  in  (2.1)  we  make  the  transformation  z  -  Xy,  we  obtain 
using  (2.2), 


y'(O)  -  X“1c 

y'  (t)  -  A  y(t)  +  yrlt(t) 


(2.3) 


or,  componentwise. 


n 


yi(t)  -  \y^t)  +  z  aljfj(t) 

4J  "  * 


(2.4) 


where  the  a's  are  the  elements  of  X”1.  Solving  for  y^(t),  we 
obtain 


A.t  A.t 

y^t)  -  yt(0)e  1  +  e  1  yo 


Xjt  X .  t  *.  — X .  s  n 

1  1  2  a^fjfsjds  .  (2.5) 

J  *  1 


Since  z(t)  »  0  is  equivalent  to  y(t)  ■  0,  we  wish  to  find  the 
least  t  for  which,  for  some  f,  y^(t)  »  0,  1  »  l,...,n,  i.e.,  for 
which 


— y  i  ( o ) 


8  n 


J-l 


aljfj(S)<l8  , 


i  -  1, . . . ,n  (2.6) 


for  some  f. 


Our  first  observation  Is  that,  given  any  starting  value 
y(O)  +  0  there  exists  a  t  >  0  and  an  f,  such  that  (2.6)  is 
satisfied.  In  fact,  there  is  a  constant  vector  f(s)  -  k  which 
does  the  trick  for  some  t  sufficiently  large.  For  substituting 
fj(s)  -  kj  in  (2.6),  we  obtain 


n 

Z 

J-l 


°ukj 


y^o) 


-s? 


-X.s 
e  ds 


^y^o) 

r 1 

?  1  -i 


whence,  by  virtue  of  the  definition  of  the  a 'a, 


Z  x 


Ji 


Vifo) 

e  1  -1 


(2.7) 


Since  — X^  >  0  the  right  member  of  (2.7)  can  be  made  as  small 
in  magnitude  as  we  please  for  sufficiently  large  t,  and  hence 
we  can  insure  that  (kj|  <£  1. 


For  each  t  }  0  we  have  a  linear  mapping  pt  taking  f  into 


the  n-dimenslonal  vector  with  i~  component 

s* e_v  *  oufj(s)ds  * 


(2.8) 


and  this  mapping  clearly  takes  our  basic  convex  set  of  f's  onto 
a  convex  subset  C(t)  of  euclidean  n— space.  For  any  f  in  our 
basic  set  there  is  another,  7,  in  the  set  which  agrees  with  f 
for  s  <£  t  and  vanishes  for  s  >  t,  so  that,  for  t'  >  t, 

Pt,7  -  pt7  »  ptf,  by  (2.8),  and  ptf  is  in  C(t')>  Thus  C(t) 


increases  with  t. 
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Now  our  desired  least  time  is,  by  (2.6),  the  least  t  ^  0  for 

which  C(t)  contains  the  vector  -y(0).  Since  C(t)  increases,  we 

have  an  interval  (t0#®)  for  which  C(t)  contains  this  vector,  while 

for  t  <  t  this  is  not  the  case.  We  cam  see  that  C(t )  also  con— 
o  'o 

tains  this  vector  as  fellows. 

Denoting  for  any  vector  x  -  (x^,...,xn)  the  euclidean  norm 
(Zx2)1/2  by  | |x| I  we  have  from  (2.8)  a  constant  k  for  which,  for 
every  f  ,t,  t',<£tQ  j  ( ,  f  I  i  k  1 1 — t '  | ;  thus  for  1 1 — t  *  |  small 
every  point  of  C(t')  is  close  to  a  point  of  C(t).  Since  -y(0) 
is  in  C(t)  for  all  t  >  tQ,  -y(0)  must  be  at  zero  distance  from 
C(tQ)  so  that  if  we  show  this  set  is  closed  — y(0)  must  actually 
be  in  it.  But  each  C ( t )  is  closed,  since  by  a  well  known  fact 
about  Banach  spaces  (*>] ,  our  basic  set  of  f's  may  be  topologized 
so  a3  to  be  compact  and  render  each  pt  continuous.  Thus  C(t), 
as  the  continuous  image  of  a  compact  set,  is  compact,  hence 
closed. 

Let  us  return  to  the  fact  that  -y(0)  is  not  in  C(t)  for 

t  <  tQ.  Prom  the  theory  of  convex  sets  [6]  this  implies  that 

we  have  a  vector  of  unit  norm,  for  which,  in  the  usual  inner 

product  notation,  (Ofc,  p^f)  £  (©S  -y(0))  for  every  f.  Since 

the  vectors  of  unit  norm  are  compact  In  the  euclidean  topology, 

t 

we  may  se'ect  a  sequence  t  Increasing  to  t  for  which  0  n  con¬ 
verges  to  some  vector  0  of  unit  norm.  But  since  p  +  f  converges 

t  t 

to  pt  f.  (0.  pt  r)  -  llra(0  n,  p  f)  £  ilm(0  n,  -y ( 0 ) )  -  (0,  -y(0)). 

c  o  n 
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Thus  If  f#  denotes  an  f  for  which  f*  -  -y(0)  we  have 

zo 

(G,  pt  f)  <£  (O,  pt  f#)  for  all  f,  hence  constants 
o  o 

not  all  zero  for  which  f*  maximizes  the  expression 


£ 

i 


VS" 


8 


{2 

i 


— X .  3 

®lalje  )fj(s)da 


(2-9) 


But  this  expression  clearly  has  as  its  maximum 


5^t#i  *  Vije_vidB 


(2.10) 


achieved  by  setting  fj(») 


-X. 


that  fj(s)  “  sgn(Z  ^aije 


-X.  o 

Z  ®iaije  /  0. 


— X .  3 

sgn(£  ).  Thus  it  is  clear 

a 

)  almost  everywhere  on  the  set  where 


Our  principal  result  now  follows,  namely  that  we  can  achieve 

minimal  time  by  restricting  f  to  assume  componentwise  +1  on  a 

finite  number  of  intervals;  in  i’act,  in  the  case  considered, 

each  component  need  change  sign  at  most  n-1  times.  This  latter 

statement  is  a  simple  consequence  of  the  fact  that  unless  the 

n  -X.s 

continuous  function  ^  given  by  ^(s)  =  2  ^laije  *3 
Identically  zero  (in  which  case  it  makes  no  difference  as  to  our 
choice  of  fj),  it  can  have  at  most  n— 1  real  zeros.  This  is  well 
known  and  there  is  a  simple  inductive  proof. 
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5 3.  A  special  case  of  n  «  2 

Consider  the  problem  as  before,  with  A 


thus 


a1  «■  — —  2z  2  +  fi 
1 

z*  «  Z|  +  fa 

a 


The  transformation 


(3.1) 


zi  -  2yt  -  y2 
z«  -  -  yi  +-ya 


reduces  the  above  system  to 


y,  -  -2yi  +  f,  +  fa 


ya  «  -yz  +  fi  +  2fa 


(3-2) 


(3.3) 


and  we  obtain,  as  before,  for  the  set  of  admissible  starting 
values,  for  a  given  t  and  f x  ,  f z , 


— y t  ( 0 )  =^t  e  (Ms)  +  f2(s))ds, 
~yz(0)  -ff  e  (ft  (.-■)+  2f2(s))ds. 


(3.'0 


« 

From  the  preceding  section,  we  know  that  if  t  Is  minimal, 
then  the  optimal  f*  Is  given  by 


2b  s 

f , ( s)  -  sgn  (ft ,e  +  ©2?  ) 


2  s  r 

f 2  (  r  )  -  sgn  (ftje  2<^e  ) 


(3.5) 
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If  wp  now  ask  the  question  "For  what  set  of  starting  values  y 
Is  It  optimal  to  choose  fi  ■  1,  f2  «  1  on  an  Initial  Interval?” 
with  a  similar  question  for  the  other  combinations  +1 ,  It  la 
readily  seen  that  the  answers  will  determine  an  optimal  policy. 
This  Is  clear,  since  any  continuation  of  an  optimal  policy  must 
be  again  optimal  with  respect  to  the  new  starting  values.  We 
thus  have 


r  2b  r  2s  2  2s  s') 

-yi (0)  -  J  e  |  sgn(ate  ♦  ^  o  )  4-  sgn  (G,e  4-  2^e  )  >  ds 


0*6) 


2  s  8 


— ya(0)  -  J  e  \  sgn e  4-  ^e  )  4-  2  sgn^e  4-  20?e  )(  d: 


To  answer  the  first  question,  for  what  values  of  y  is  It  opti¬ 
mal  to  set  f|  ■  f2  ■  1  on  an  Initial  Interval,  we  note  tnat 
this  Is  equivalent  to  the  conditions 

t*  >  0 

+  >  0  (3*7) 

dj  4-  2©2  5  u 

2s  3  2s  s 

Now,  since  tne  functions  4-  ^e  ,  4  l^e  Can 

e  i  h  vanish  at  most  once,  we  see  that  the  above  case  breaks  down 
Into  four  sub— cases,  namely: 


p  -5*52 


• 

2t 

• 

t 

(a) 

©l  e 

♦  »*e  >  0, 

• 

• 

2t 

t 

e 

+  2d^  e  >  0 

(b) 

V 

A 

(5.8) 

U) 

A 

V 

(d) 

<,  < 

Ca  se 

(a)  la 

trivial  and  consists  of  the  arc  a*  illustrated 

figure 

1  .  a* 

is  defined  parametrically  by 

2t 

y  i  ('->)  -  1  -  * 

t* 

yc  ( )  ■  Ml—'1  )» 


t  >  o 


o.<?) 


as  onp  can  readily  verify  by  worKlng  out  the  Integrals.  More¬ 
over,  the  curve  defines  an  optimal  path,  since  the  solution  of 
the  differential  equation  is,  with  fj  -  f2  -  1  identically,  and 
y i  ( C )  ,  y-(0)  defined  a  above,  precisely  a  sub-arc  of  a'  beginning 
at  y  (o)  and  terminating  at  the  origin. 

'ace  (b)  is  vacuous,  for  if  we  have 


2 1 # 

t* 

e  + 

Ose 

>0  and 

2t  * 

t* 

2^e 

<  o, 

(3.10) 


wi*  obtain,  by  subtraction,  and  tne  condition  t*  >  0,  tn.it  Cs  <  0. 
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But,  ©•!  ♦  Q?  >0,  whence  £|  >  0.  We  thus  have 

2 1*  t»  t*  t*  t* 

Qte  ♦  20ae  >  £te  ♦  2ttae  -  e  (£1+2^2)  >  0 


which  contradicts 

2t*  t* 

UjP  +  2^e  <  0 


We  shall  treat  case  (3.0c)  In  detail.  Case  (j.bdj 
treated  similarly,  but  Is  a  trifle  more  involved,  albdt 
and  will  be  omitted  on  those  groundB. 

We  nave,  upon  substituting  in  (,5.6)  for  case  (c): 


-y. (o) 


ln(-»s/«i ) 


o 


2s 

e  ds 


In ) 


t*  0 

c'S  n  2s 

e  d '  +  j  e 

O 


In  (-**/»»  ) 

-ys(o)  -  J 

o 


8 

d  ? 


f 


e  ds 


1  n  ( — ©v  ) 


O 


d: 


Simplifying,  we  obtain 

-y  i  (0)  -  (£a  An  )2  -  1 

-Yz  ( 0 )  -  -  2  (©s/d, )  ♦  et#  -j 

t  * 

If  now  we  set  x*  -  e  ,  our  conditions  become 


(3.11) 


(3.12) 

car.  be 
elementary , 


da 

0.15) 


(5.  H) 
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x*  >  1 

©I  ♦  *2  >  0 

£i  +  >  0 

Qi x*  ♦  Qs  <  o  ^  -j , 

0,x*  ♦  20a  >  0 
yi (o)  -  l  -  («s/©i)* 
y*(o)  -  ;  *  2(o»/ei )  -  %• 

We  eaaliy  obtain  from  the  above  that  <  0.  By  homogeneity, 
we  can  set  ^  »  —  1,  &s  “A  and  we  obtain  the  equivalent  condi¬ 
tion:- 


>  x*  >  \  >  1 


(A) 


y i  (o)  -  i  -  A2 
y*(o)  -  3  -  2  A  -  X* 


(B) 


l.e.,  we  wleh  to  find  the  Image  of  all  pair::  ,x#,A)  satisfying 
(A)  unler  the  mapping  defined  by  ( B ) .  Plctorlally 
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On  the  ot.ner  hand,  the  Jacobian  of  the  transformation  (B)  la 
given  by 


J 


B 


-2^  0 
-2  -1 


2A  f  0 


0-16) 


throughout  (A  ) ; 
boundary  of  the 
of  this  fact  we 


hence  the  t ran;; f orma t ion  lc  non-" lnguiar  and  the 
image  1:;  the  Image  of  th'4  boundary  Maying  use 
obtain  the  region  for  case  (3 .do): 


y.  (o)  <  o 


(J.in 


and 


)-Vj  i  — y i  (o)  <  -jA 0)  <  J  -  i-yi(oT  (3.  id) 

In  a  similar  manner  we  obtain  a  region  for  case  (*.8d).  The 
union  of  cases  (3.3a)  tnrough  (3.3d)  lr  the  set  cf  all  starting 
values  for  which  fi  *  f*  •  1  Is  optimal  on  an  Initial  Interval. 

In  a  similar  manner  we  obtain  tne  region  l'j  *  1,  f.  -  -  1. 

(Notice  t.nat  we  need  not  compute  the  oth^r  regions  Ince  they  can 
be  obtained  by  skew— symme try  .  ) 

The  final  result  of  our  calculations  is  Illustrated  In 
Figures  1  and  2.  Figure  2  Is  the  Image  of  Figure  1  under  our 
Initial  transformation  and  gives  the  optimal  policy  In  terms  of 
our  Initial  starting  vector  c  -  (zi  (0),  za  ( 0 ) ) . 


P-552 

-13- 


Ir.  terms  of  optimal  paths  (cf.  Fig.  2)  wo  can  state  the 
following:  A  path  initiating  in  the  (1,1)  region  continues 
with  fi  -  1,  f?  »  1  until  it  strikes  either  the  straight  seg¬ 
ment  SB  nr  the  parabolic  arc  p.  In  the  former  case  ft  switches 
from  1  to  —1  and  the  path  continues  along  £55  to  the  origin.  In 
the  latter  ease  fi  switches  to  -1  at  p  and  the  psth  continues  in 
the  (—1,1)  region  until  it  intercepts  the  parabolic  arc  a  at 
which  fa  changes  from  1  to  -1  and  a  is  follcv;od  to  the  origin 
with  f!  -  f 2  ■  — 1 .  Similar  remarks  hold  for  the  skew-cymmetrlc 
regions . 
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